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Abstract — This paper concerns the capacity of the discrete 
noiseless channel introduced by Shannon. A sufficient condition 
is given for the capacity to be well-defined. For a general discrete 
noiseless channel allowing non-integer valued symbol weights, it 
is shown that the capacity — if well-defined — can be determined 
from the radius of convergence of its generating function, from 
the smallest positive pole of its generating function, or from the 
rightmost real singularity of its complex generating function. 
A generalisation is given for Pringsheim's Theorem and for 
the Exponential Growth Formula to generating functions of 
combinatorial structures with non-integer valued symbol weights. 

I. Introduction 

When modelling digital communication systems, there are 
situations where we do not explicitly model physical noise. 
We rather introduce constraints on the allowed system config- 
urations that minimise the influence of undesired effects. An 
example is the runlength-limited constraint in magnetic record- 
ing |H1. We consider in this paper the discrete noiseless channel 
(DNC) as introduced by Shannon [2 ]. A DNC is specified by 
a set of constraints imposed on strings over a certain alphabet, 
and only those strings that fulfil the constraints are allowed 
for transmission or storage. A DNC allows the specification 
of two types of constraints. The first constraint is on symbol 
constellations (for example, only binary strings with not more 
than two consecutive Os are allowed), and the second constraint 
is on symbol weights (for example, the symbol a has to be of 
duration 5.53 seconds). Depending on the system we want to 
model, the symbol weights represent the critical resource over 
which we want to optimise. This can for example be duration, 
length or energy. We then ask the following question. What 
is the maximum rate of data per string weight that can be 
transmitted over a DNC? 

This question was first answered by Shannon in Q. In Q, 
the authors extend Shannon's results to DNCs with non-integer 
valued symbol weights. In both J2] and Q, the authors use 
the following approach to derive the capacity of a DNC. They 
restrict the class of considered DNCs to those that allow the 
transmission of a set of strings forming a regular language. 
The regularity allows then to represent the DNC by a finite 
state machine and results from matrix theory are applied to 
derive the capacity of the DNC. 

Our approach is different in the following sense. We con- 
sider general DNCs with the only restriction that the capacity 
as defined in |2| and generalised in O has to be well-defined, 



which will turn out to be a restriction on the set of possible 
string weights. 

This allows us then to represent the combinatorial complex- 
ity of a DNC by a generating function with a well-defined 
radius of convergence and we use analytical methods to derive 
the capacity. In this sense, our work is a generalisation of Q. 
Perhaps more important, in many cases that could be treated by 
the techniques proposed in Q, it is much simpler to construct 
the generating function of the considered DNC and to use our 
results to derive the capacity. We give two simple examples 
that may serve as illustrations. In this sense, our work can also 
be considered as an interesting alternative to 0. 

II. Definitions 

We formally define a DNC and its generating function as 
follows. 

Definition 1. A DNC A = (A,w) consists of a countable 
set A of strings accepted by the channel and an associated 
weight function w: A i— > R® (R® denotes the nonnegative 
real numbers) with the following property. If Si,s 2 € A and 
sis 2 € A (sis 2 denotes the concatenation of si and s 2 ), then 

w(sis 2 ) = w(si) + w(s 2 ). (1) 

By convention, the empty string e is always of weight zero, 
i.e., w(e) = 0. 

Definition 2. Let A = (A, w) represent a DNC. We define 
the generating function of A by 

We order and index the set of possible string weights w(A) 
such that w(A) = {wk} k % 1 with w\ < w 2 < ■ • • . We can 
then write 

00 

G A ( y ) = J2 N b»k}y Wh (3) 

k=l 

where for each k £ N, the coefficient N[wk] is equal to the 
number of distinct strings of weight Wk- Since the coefficients 
A^w/j] result from an enumeration, they are all nonnegative. 
Note that for any DNC A, we have G^(0) = iV[0] = 1 since 
every DNC allows the transmission of the empty string and 
since there is only one empty string. 



The maximum rate of data per string weight that can be 
transmitted over a DNC is given by its capacity. We define 
capacity in accordance with Q and as follows. 

Definition 3. The capacity C of a DNC A — (A, w) is given 
by 

\nN[w k ] 

C=limsup (4) 

fc^oo Wk 

in nats per symbol weight. This is equivalent to the following. 
For all e with C > e > 0, the following two properties hold. 

1) The number N[wk] is greater than or equal to e Wk ^ c ~^ 
infinitely often (i.o.) with respect to k. 

2) The number N[wk] is less than or equal to e Wk< ^ c+e ' 
almost everywhere (a.e.) with respect to fc. 

We assume in the following that the number sequence 
{ v °k}'kLi is not too dense in the sense that for any integer 

n > 

maxfc < Ln K (5) 

w k <n 

for some constant L > and some constant K > 0. Otherwise, 
the number of possible string weights in the interval [n, n+1] 
increases exponentially with n. In this case, Definition [3] does 
not apply. We present in the following example a case where 
capacity is not well-defined. 

Example 1. Let N[w k ] denote the coefficients of the generat- 
ing function of some DNC. Assume N[wk] = 1 for all k S N 
and assume 

maxfc = \R n ] (6) 

w k <n 

for some R > 1. According to Definition [3] the capacity of 
the DNC is then equal to zero because of \nN[wk] = for 
all fc e N. However, the channel accepts R n distinct strings 
of weight smaller than n. The average amount of data per 
string weight that we can transmit over the channel is thus 
lower-bounded by hxR n /n = lni?, which is according to the 
assumption greater than zero. <a 

Whenever we say that the capacity of a DNC is well-defined, 
we mean that © is fulfilled. 

III. Capacity by Radius of Convergence 

One way to calculate the capacity of a DNC is by determin- 
ing the radius of convergence of its generating function. 

Lemma 1. Let A be a DNC with the generating function 
G A (y). If the capacity C of A is well-defined, then it is given 
by C = — In R where R denotes the radius of convergence of 

G A (y). 

In the proof of this lemma, we will need the following result 
from 0. 

Lemma 2. If © is fulfilled and if p is a positive real number, 
then X^fc^Li P Wk converges iff p < 1. 



Proof of LemmaU} We define M[k] = N 1 / Wk [wk] and 
write G A (y) as 

oo 

fc=i 

We define the two sets D(y) and E(y) as 

D(y) = {ke N\M[k]y < l} (8) 

E(y)=n\D(y) = {keN\M[k]y>l} (9) 

and write 

G A (y) = £ {M[k]y) Wk + £ (M[l]y) w '. (10) 

keD(y) leE(y) 

It follows from Lemma [2] that G A (y) converges iff the set 
E(y) is finite. The number R is the radius of convergence 
of G A (y), therefore, for any 8 > 1, the set E(R/S) is finite. 
Since D(y) = N\E(y), the finiteness of E(R/S) is equivalent 
to 

keD(R/S) a.e. (11) 
We define e = In 6. Equation (fTTT i is then equivalent to 

N[w k ] < e ™*(-lnJJ+«) ae (12 ) 

which implies 

N[w k ] <e Wk ^ lnR+e) a.e. (13) 

Again since R is the radius of convergence of G A (y), for any 
5 > 1, the set E(RS) is infinite. For e = ln<5, this is equivalent 
to 

N[w k ] > e Wk{ - XnR ~^ i.o. (14) 

It follows from (O and (O and Definition [3] that - In R is 
equal to the capacity of A. We therefore have C = — In R. ■ 
In the following example, we show how Lemma [T] applies 
in practice. We denote by A U B the union of the two sets 
A and B, we denote by AB the set of all concatenations ab 
with a 6 A and b € B, and we denote by S* the Kleene star 
operation on S, which is defined as S* = e U S U SS U • • • . 

Example 2. We consider a DNC A = (A, w) with the alphabet 
{0, 1} and symbol weights w(0) = 1 and w(l) = tt. The DNC 
A does not allow strings that contain two or more consecutive 
Is. We represent A by a regular expression and write A = 
{eUl}{0U01}*. For the generating function of A we get 

oo 

G^a/) = (1 + 2T)£(2/ + Z/ 1+ T l . d5) 

n=0 

The radius of convergence is given by the smallest positive 
solution R of the equation y+y 1+7r = 1. We find R = 0.72937. 
According to Lemma Q] the capacity of A is thus given by 
C = -lni? = 0.31558. < 



IV. Capacity by Rightmost Real Singularity 



is determined by 5ft {s} > Q. For F A (e s ), we have 



There are cases where we derive the closed-form represen- 
tation of the generating function of a DNC without explicitly 
using its series representation. The techniques introduced in 
101 and J5] may serve as two examples. In this section, we 
show how the capacity of a DNC A can be determined from 
the closed-form representation of its generating function. We 
do this in two steps. We first identify the region of convergence 
(r.o.c.) of the complex generating function F^(e _s ) with 
its rightmost real singularity. The complex generating func- 
tion F^(e~ s ) results from evaluating the generating function 
G A (y) in y — e~ s , seC. Second, we show that the rightmost 
real singularity of F A (e~ s ) determines the capacity of A. 

Theorem 1. If the r.o.c. ofF A {e~ s ) is determined by 5ft {s} > 
Q, then F^(e _s ) has a singularity in s = Q. 

Proof: Suppose in contrary that F^(e~ s ) is analytic in 
s = Q implying that it is analytic in a disc of radius r centred 
at Q. We choose a number h such that < h < r/3, and we 
consider the Taylor expansion of F^(e~ s ) around sq = Q + h 
as follows. 

^(o = E [F ^ (e T )]( ^ -*°)" < 16 > 

— ' n\ 

oo 

oo E N{w k ](~w k ) n e- W « s ° 
= E~ i (*-*o) n . (17) 

For s = Q — h, this is according to our supposition a converg- 
ing double sum with positive terms and we can reorganise it 
in any way we want. We thus have convergence in 



E N[w k ]{-w k ) 



n=0 



w?(2h) n 



k=i 



n=0 



e ~w k so e w k 2h 



k=l 



k=l 



w k \e 



-w k (Q-h) 



{-2h) n (18) 
(19) 
(20) 
(21) 



But convergence in the last line contradicts that the r.o.c. of 
F A (e~ s ) is strictly given by 5ft {s} > Q. ■ 
We now relate the rightmost real singularity of F^(e _s ) to 
the capacity of A. 

Theorem 2. Assume that F_4(e~ s ) has its rightmost real 
singularity in s = Q. The capacity of A is then given by 
C Q. 

Proof: Since F_4(e _s ) has its rightmost real singularity in 
s = Q, it follows from Theorem Q] that the r.o.c. of F^(e _s ) 



F A (e- s ) = J2N[w k ]t 



k=l 
oo 

< J2 \ N i w k]e 



k=l 



w k \\e 



(22) 
(23) 
(24) 



fe=i 



where equality in (1241) holds because the coefficients N[w k ] are 
all nonnegative and where we have equality in (f23b if s is real. 
It follows that if the r.o.c. of F A (e~ s ) is given by 5ft {s} > Q, 
then the radius of convergence of G^(y) is given by R — . 
Using Lemma [U we have for the capacity C = — \nR = Q. 

■ 

Note 1. With Theorem [T] and Theorem we generalised 
Pringsheim's Theorem and the Exponential Growth Formula, 
see 0, to generating functions of DNCs with non-integer 
valued symbol weights. 

V. Capacity by Smallest Positive Pole 

We formulate the most important application of Theorem [2] 
in the following corollary: 

Corollary 1. Let A represent a DNC with a well-defined 
capacity C. Suppose that the generating function G A (y) can 
be written as 



G A (y) 



my 



n 2 y 



diy Vl + d 2 y v 



d q y u i 



(25) 



for some finite positive integers p and q. The capacity C is 
then given by — In P where P is the smallest positive pole of 
G A (y). 

Note 2. The corollary was already stated in |4l Theorem 1]. 
However, the proof given by the authors does not apply for 
the general case, which we consider in this paper. 

Proof of Corollary [7} If G A (y) is of the form (T25l ), 
the complex generating function F^(e~ s ) as defined in the 
previous section is meromorphic, which implies that all its 
singularities are poles. The substitution y = e~ s , for s real, is a 
one-to-one mapping from the real axis to the positive real axis. 
Therefore, if Q is the rightmost real singularity of F^(e~ s ), 
then e _< 5 is the smallest positive pole of G A (y). Applying 
Theorem [2] we get for the capacity C = Q = — In P. ■ 

Example 3. We consider the DNC A = (A, w) where A is 
the set of all binary strings that do not contain the substring 
111 and where the symbol weights are given by w(0) = w(l). 
We use a result from in the form of E Proposition 1.4]. 
It states that the set of binary strings that do not contain a 
certain pattern p has the generating function 



M = 



c{y) 



y k + (1 - 2y)c(y) 



(26) 



where k is the length (in bits) of p and where c(y) is the auto- 
correlation polynomial of p. It is defined as c(y) — Y^i^o Ci y % 
with q given by 

Ci = 8[p 1+i p 2+ i ■ ■ -p k ,PiP2 ■ ■ -Pk-i] (27) 

where pi denotes the zth bit (from the left) of p and where 
S[a, b] = 1 if a = b and S[a, b] — if a ^ b. For p = 111, 
we have c(y) = 1 + y + y 2 and k = 3. This yields for the 
generating function of A 

G-(W = 3 T n — x n , t 2\ - (28) 
y J + (l-2y)(l + ?/ + y- i ) 

Note that the application of the technique from [4 1 would have 
led to the same formula. For the smallest positive pole P of 
Ga(u) we nn d P = 0.54369. According to Corollary Q] the 
capacity of A is thus given by C = — In P = 0.60938. < 

VI. Conclusions 

For a general DNC, we identified the capacity with the 
characteristics of its generating function, namely the radius 
of convergence of its generating function, the rightmost real 
singularity of its complex generating function, and the small- 
est positive pole of its generating function. We generalised 
Pringsheim's Theorem and the Exponential Growth Formula 
as given in (6) to generating functions that allow non-integer 
valued symbol weights. 

Representing a DNC by its generating function and not by 
a finite state machine has an additional advantage. Although 
the finite state machine allows the derivation of the correct 
capacity of the DNC, it says nothing about the exact number 
of valid strings of weight w. The generating function of a DNC 
provides this information. The coefficients N[wk] are equal to 
the number of distinct strings of length that are accepted 
by the DNC. The coefficients can either be calculated by an 



algebraic expansion of the generating function or they can be 
approximated by means of analytic asymptotics as discussed 
for integer valued symbol weights in |6|. In Q, the analytic 
approach is extended to generating functions of DNCs with 
non-integer valued symbol weights. 

For a regular DNC fulfilling some further restrictions, the 
authors in [3 | define a Markov process that generates valid 
strings at an entropy rate equal to the capacity of the channel. 
Based on generating functions as introduced in this paper, it 
is shown in |7| that for a general DNC, any entropy rate C 
smaller than the capacity C is achievable in the sense that 
there exists a random process that generates strings that are 
transmitted over the channel at an entropy rate C. 
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